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SUBSTITUTE SPECIFICATION 

METHOD FOR MAKING A COLORFUL 3D MODEL 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates to a method for constructing a three-dimensional model, 
and more particularly to a method using the transformation of a generic model with regular 
mesh structure embedded therein to have the regularly transformed mesh structure. Further, the 
method is able to automatically compensate color difference between two adjacent transformed 
mesh to reach a highly realistic surface effect. 

2. Description of Related Art 

[0002] Nowadays, a computer-generated three-dimensional (3D) model has been widely used 
in different fields, i.e. from the characters in video games to special visual effect in the movie 
industry or from the commercial development of multi-media to special requirements in the 
medical industry. As a consequence, the construction and operation of 3D data or 3D model 
have become cmcial lessons in the field of making a 3D model. 

[0003] The conventional way of making a 3D model starts from the drafting of the animation 
engineer by using the modeling software. Normally, it took a long time to train a qualified 
animation engineer. After the animation engineer is qualified, the engineer still has to use 
creativity to add in "personal touch" in the modeling process and also the coloring to the 
finished model to make the model as perfect as possible. This art creation process takes a long 
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time to finalize the entire process. Also, the "personal touch" sometimes may become the 
greatest failure in the entire creation process. 

[0004] In contrast to the conventional model creation method, using measurement devices to 
construct a 3D model belongs to the category of reverse engineering. The shape and color 
information can be retrieved by using deUcate devices with 0.01cm or higher accuracy. The 
measured data of the shape and color of an object usually is presented by a triangular mesh or 
curved surface to show the geometry information, which is shown in Fig. 1 A. A 
two-dimensional image is shown in Fig. IB to show the color information. The 
interrelationship between the color information and the geometry information is shown by 
texture mapping. The mapping is often referred to as texture coordinate. Li order to have a 
complete model, a different measurement angle to the object is required. Then the measured 
data is adjusted and integrated into the same spatial coordinate system, which is shown in Fig. 
IC. Thereafter, the data is integrated into a complete 3D model as shown in Fig. ID. 

[0005] The model created by reverse engineering process has high accuracy in relation to the 
object. The difference is hardly to be recognized by the naked eye. Besides, there is no special 
training program required for the operator. The operator only needs to be familiar with the 
equipment. However, the data obtained fi-om the measurement instmment usually is enormous 
and lacks regularity, which results in that the data can only be used in the production of a 
specific object. Besides, the large quantity of data hinders the post-process, e.g. data 
transmission or data reproduction. Furthermore, as an influence from the light, the data from 
different measuring angles has obvious color difference. Therefore, a complete method to 
practically use the original measured data is required to solve the previously described 
problems. 



2 



[0006] To correct the problems, it is recommended to construct a 3D model by using special 
tools. Still the time spent for manually constructing a 3D model does not meet the 
cost-effectiveness requirement. Due to the fast growth of reverse engineering, a highly precise 
measurement instrument is applied to retrieve an object's 3D data to recreate a vivid model out 
of the object measured. 

[0007] U.S. Pat. No. 6,512,518 (the '518 patent) discusses a method of using a laser scanning 
device to retrieve object 3D data and then the obtained data is transformed into meshed data. A 
method for integrating the meshed data is also provided. The '518 patent is able to quickly and 
accurately measure the spatial position of an object so that a highly accurate model is produced. 
However, the spatial position is represented by a densed dot group data, which is large and 
irregular. Consequently, the re-use of the measured data is highly unlikely. U.S. Pat. No. 
6,356,272 ('272 patent) applies shape from a silhouette principle to utiUze a fixed camera 
system to take large numbers of pictures to create a 3D model from the continuous images and 
establish the mapping relationship between the images and the mesh. The pictures taken by the 
'272 patent are continuous from the sides of the object. The best mapping relationship is 
chosen from the angle between the normal of a triangle and the image. The top and bottom of 
an object or an object with a complex appearance may have data distortion when mapping 
occurred. 

[0008] To overcome the shortcomings, the present invention tends to provide an improved 
method to make a vivid and colorful model to mitigate the aforementioned problems. 

SUMMARY OF THE INVENTION 

[0009] The primary objective of the present invention is to provide an improved method which 
is able to integrate the retrieved data into complete colorful information so as to establish a 
vivid 3D model. Besides, the retrieved data from an object is mapped to a generic model having 
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regular data embedded therein. After mapping, the data from the generic model is forced to 
transform into usable, regular geometry information for the model. 

[0010] Another objective of the present invention is to provide a color mending method to 
compensate for color difference between adjacent data such that the surface color on the model 
is smooth and continuous. 

[0011] Other objects, advantages and novel features of the invention will become more 
apparent from the following detailed description when taken in conjxmction with the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] Fig. 1 Ais a schematic view to show the geometry information of a picture by using the 
triangular mesh or a curved surface; 

[0013] Fig. IB is a schematic view showing the color information of a two-dimensional image; 

[0014] Fig. IC is a schematic view showing the integration of all the measured data into the 
same spatial coordinate; 

[0015] Fig. ID is a schematic view showing that all the measured data is integrated into a 
complete three-dimensional model; 

[0016] Fig. 2 is a flow chart showing the production of the 3D model; 

[0017] Fig. 3 A is a schematic view showing the original mesh as measured by the measuring 
instrument; 

[0018] Fig. 3B is a schematic view of mesh of a new model with precise and regular data; 
[0019] Fig. 3C is a schematic view showing color difference between adjacent mesh; 
[0020] Fig. 4 is a flow chart of reconstructing regular mesh model; 
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[0021] Fig. 5 is a schematic view showing the original mesh and the corresponding color 
information; 

[0022] Fig. 6 is a schematic view of the selected generic model; 

[0023] Fig. 7 is a schematic view of the transformed generic model; 

[0024] Fig. 8A is a schematic view showing the original measured data; 

[0025] Fig. 8B is a schematic view of the reconstructed model by using the generic model of 
Fig. 6; 

[0026] Fig. 9 is a schematic view showing that the texture image data is extracted from the 
original measured data; 

[0027] Fig. 10 is a flow chart of abstracting color map information; 

[0028] Fig. llAis a schematic view showing the spatial interrelationship between the texture 
image data of the original measured data and the generic model; 

[0029] Fig. 1 IB is a schematic view showing that the texture image data is reattached to the 
generic model to complete the color abstracting process; 

[0030] Fig. 12 is a flow chart showing the harmonization of color between two measured 
mesh; 

[0031] Fig. 13 is a schematic view of the overlapping relationship and the arrangement 
sequence of the measured data; 

[0032] Fig, 14 is a schematic view showing that the overlapped portion of two adjacent mesh 
models and the overlapped portion correspond to respective textxire image; 

[0033] Figs. 15 A and 15B are a comparison between the mesh models before and after 
adjustment of color average; 
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[0034] Fig. 16 is a flow chart showing the pixel blending; and 

[0035] Figs. 17 A, 17B and 17C are schematic views showing the advanced comparison resuh 
from Figs. 15A and 15B. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0036] The present invention relates to a method of processing three-dimensional (3D) data to 
integrate the measured 3D data from the object to be reproduced into a complete 3D color 
model. In the geometry information aspect, the method applies a generic model to combine 
measured data from different angles of the object to become a mesh model with regular data 
embedded therein, 

[0037] In the color information aspect, the method applies the comparison between the spatial 
corresponding relationship of the newly produced regular data of the mesh model and the 
original measured data to reattach the texture image data of the measured data to the model. 
The color difference between adjacent images is then adjusted so that by means of interactive 
measurement, the operator is able to easily construct a 3D model with high accuracy and 
applicability. 

[0038] The method uses a generic model to circumstantially integrate the original measured 
data into a complete model. The word "generic" means that the model is applicable to all sorts 
of objects with the similar appearances such that severe distortion may be avoided. For 
example, to construct a hiraian head model, a generic model with facial characteristics, i.e. a 
pair of eyes, a nose, a mouth and a pair of ears, may be applied. To construct an animal such as 
a cow, a horse or even a sheep, a generic model with four legs may be applied. 

[0039] The original and vast mesh of the object is not dealt with in the present invention but to 
adopt the pre-designed generic model with regular mesh stmcture to map with the original 
measured data such that a rough model with the same appearance as that of the object being 
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measured is built. If there is any data breakage such as hair or other parts of the object that are 
not easily measured, the breakage may be mended automatically by applying the mesh 
structural relationship between adjacent data in the mapping process. The corresponding 
relationship of the texture images is automatically established by using the spatial relationship 
between the generic model and the measured data without the involvement of special 
positioning equipment or any manual operation. 

[0040] The method of the present invention mainly is divided into four major parts: 
reconstructing a regular mesh model; 
abstracting color; 

harmonization of color arrangement; and 
pixel blending between overlapped images. 

[0041] With reference to Fig. 2, the first step of the present invention is to reconstruct a regular 
mesh model. The data measured by the three-dimensional measuring device is dense, as shown 
in Fig. 3 A, to reduce the error by replacing the curved surface model with the mesh model. This 
is particularly true for those objects with complicated shapes or fine minute characteristics. 
However, the more accurate the measured data is, the larger quantity of the triangular mesh it 
becomes. Therefore, direct implication of the original measured data may lead to a 
consequence that the quantity of the meshes is large and is not applicable. 

[0042] Therefore, it is recommended to use a generic model with regular mesh structure 
embedded therein to map with the original measured data to generate a new model. The new 
model has a regular mesh structure inherited fi-om the generic model; meanwhile, it is 
transformed to a shape similar to the original measured data, as shown in Fig. 3B. Further, due 
to the overlapping relationship in the spatial positions between the original measured data and 
the data of the new model, the texture images firom the original measured data are projected to 
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the new model so as to establish the corresponding relationship between the new model and the 
texture images. 

[0043] When the second step is finished, the construction of a complete model with regular 
mesh structure and multiple color texture images is completed. However, due to the color 
difference between the texture images taken fi-om different viewing angles, as shown in Fig. 3C, 
the overlapping areas of images are used to adjust the color difference to allow the brightness 
of all the images to become the same. Pixel blending is processed in the image overlapping area 
so that a 3D model, as shown in Fig. 3D, with concise mesh structure and smooth surface color 
is generated. 

[0044] With reference to Fig. 4, the original measured data (100) is a group of mesh models 
obtained by a 3D measuring device. Each model is composed of mesh data (110) and texture 
image data (120), obtained by measurement in different angles to an object to be reproduced. 
All of the models are then transformed into the same coordinate system. In step (SI 02), 
according to the shape of the object, a generic model (200) with a similar appearance to the 
object is selected. In step (SI 04), the generic model is roughly overlapped with the original 
measured data in space. In step (SI 06), the dimension of the generic model is adjusted to 
correspond to the dimension of the original measured data. In the last step (SI 08), the generic 
model is projected to the original model. Consequently, the data of the generic model is 
deformed and thus the generic model has an appearance similar to that of the original measured 
data (100). Even so, the data of the generic model still has a regular mesh structure 
characteristic. 

[0045] Fig. 6 shows a generic model (200) ready for use in the present invention. Fig. 7 shows 
the appearance change of the deformed generic model (210). Fig. 8 A and Fig. 8B show the 
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diflferences in the quantity of mesh and mesh distribution between the original measured data 
(100) (Fig. 8 A) and the deformed generic model (210). 

[0046] Color abstracting is used to separate texture image data (120) from the original 
measured data (100). Then the texture image (120) is re-mapped to the deformed generic model 
(210), as shown in Fig. 9. As a matter of fact, to establish the corresponding relationship 
between the deformed generic model (210) and the texture image (120), the texture coordinate 
and the corresponding texture image of each mesh point of the deformed generic model (210) 
are required. Because each mesh point of the deformed generic model (210) is projected to the 
original measured data (100), the triangle containing the projected mesh point is used to 
calculate the texture coordinate, and the texture image corresponding to the triangle of the 
deformed generic model mesh point is used as the corresponding texture image. 

[0047] With reference to Fig. 10, step (S202) is to choose the corresponding triangle of mesh 
point of the deformed genetic model (210). Step (S204) is to calculate the texture coordinate of 
the chosen triangle of the mesh point of the deformed generic model (210) to correspond,to the 
texture image. In step (S206), the continuity of the triangles chosen is checked to see if the 
chosen triangles are within the same texture image. If the coordinates of the chosen triangles 
are not continuous, not within the same texture image, other triangles should be selected to 
calculate the corresponding coordinates, which is shown in step (S208). Then to complete the 
calculation of the coordinates of all the triangles of the deformed generic model (210), in step 
(S210), a process is required to check if all the triangles of the deformed generic model (210) 
are calculated. 

[0048] With reference to Figs. 1 1 A and 1 IB, after color abstraction, the generic model (220) is 
a three-dimensional colorful model and contains multiple texture images. However, because 
the texture images are taken from different angles, color differences between the texture 



9 



images may occur. In order to harmonize the surface color of the generic model (220), the 
overlapped characteristic between the texture images is used. 

[0049] With reference to Fig. 12, step (S302) is to seek the overlapped area Oij between 
measured data (100). That is, if the measured data (100) is M={Mi, M2, M3,. . .Mn}, n 
three-dimensional mesh, Oy stands for the overlapped area between any two adjacent measured 
data Mi, Mj. In step (S304), the magnitude of Oy is determined. Then in step (S306), the 
sequence of M is determined. Therefore, if Mi is the first layer Mli, all the mesh model related 
to and overlapped with Mi is Ml2. Thus all the mesh model related to Ml2 is Mls and so on. 
Each mesh model in each layer is arranged in a descending manner according to their 
magnitudes. Thus a new three-dimensional mesh model group M' ={M' i,M' 2,,„,M' „} is 
obtained. 

[0050] Fig. 13 shows the overlapping relationship and the layer sequence of the measured data. 
Fig. 14 shows the overlapped area between two adjacent mesh models and the overlapped areas 
respectively correspond to their own texture images. 

[0051] In step (S308), according to M ' sequence, the color adjustment value Ai of the texture 
image of the mesh model is calculated in relation to the intensity average of the overlapped area 
of each mesh model, which is: 

Intensity average value of the overlapped area of M'i is: 1avq\ =1,2,3,. . .,n 

Color adjustment value of M'l Ai=l 

[0052] Then the color adjustment value of M'l influenced by M'i is Ai,i=AiX(lAVQi/lAVQi). 

[0053] Then if all the mesh models that are overlapped with M'i are taken into consideration, 
color adjustment value of M'i would be: 

AKAi,i xWi,i+. . . .+Ai,i.ix Wi,i.i)/(Wu+. . .+Wm.i) 
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where Wi is the mesh influenced weight value. 

[0054] Fig. 1 5 shows the comparison between and after color average adjustment of a group of 
mesh models, wherein Fig. 15 A is before the color adjustment and Fig. 15B is after color 
adjustment. 

[0055] Pixel blending is processed to the images in the overlapped areas to harmonize the color 
of two adjacent images. 

[0056] With reference to Fig. 16, step (S402) is to seek all the overlapped triangles and the 
texture images covered by the triangles. To triangle T, if the corresponding texture images are 
It,i,It,2. . .jT,m, the m texture images are overlapped in all the parts of Ti,i,Ti,2„„Ti,m in relation 
to triangle T. Therefore, pixel blending is processed to these overlapped mapped areas. 

[0057] In step (S404), to each triangle T in the overlapped area, calculation of the distances D 
of the vertices of the triangle T to the nearest boundary vertex is required. Because the triangle 
T has m corresponding mesh models, the distances Dl, D2. . .,Dm are obtained by calculating 
each vertex of the triangles. In step (S406), each triangle in the overlapped area is used as an 
unit so that pixel blending weight average is processed to the texture image corresponding area 
covered by the unit. To the vertex Vi(i= 1,2,3) of each triangle, the weight of the pixel blending 
is Di,i,Di^,. . ..Di,ni. The pixel color of the covered images is Ci,i,Ci^, . . .Ci^. The color after 
pixel blending is Ci,AVG. To every sampling point within the triangle, the pixel blending weight 
is calculated by applying the barycentric coordinate principle. Then the color after pixel 
blending is processed by using the same principle: 

Ci^VG=(Ci.l XDi.i+Ci^xDi;2. . .+(Ci,mXDi.„,) 

[0058] Fig. 17 is an advanced comparison to Fig. 15, wherein Fig. 17C is the result after pixel 
blending to the overlapped area of Fig. 17B. 
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[0059] The following tables illustrate the advantages of the present invention. 



"""\^^^^^Meth^ 


Conventional 
method 


LJ.S. Pat. 
6,512,518 


U.S. Pat. 
6,356,272 


Present invention 


Owner 




Cyra 


Sanyo Electric 


llKl 


Treatment 


Manual 


Interactive 


hiteractive 


Interactive 


Constructing time 


Longest 


Long 


Short 


Short 


Mesh structure 


Regular 


hregular 


Irregular 


Regular 


Texture manning 


Manual 




Auto-mapping 


Auto-mapping 


Color evenness 


Excellent 




Bad 


Excellent 


Appearance 
similarity 


Fair 


Good 


Good 


Excellent 


Reusability 


Excellent 


Bad 


Bad 


Excellent 


Others 








Auto-repair to 
data discontinuity 
(such as hair) 



[0060] It is to be imderstood, however, that even though nimierous characteristics and 
advantages of the present invention have been set forth in the foregoing description, together 
with details of the structure and function of the invention, the disclosure is illustrative only, and 
changes may be made in detail, especially in matters of shape, size and arrangement of parts 
within the principles of the invention to the full extent indicated by the broad general meaning 
of the terms in which the appended claims are expressed. 

60686443 vl 
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